Causal inference
29 bites tagged Causal inference: interview questions with model answers, and 60-second explainers.
What metrics and models link onboarding to long-term retention?
Track activation-to-D90 metrics; use Cox/AFT or diff-in-diff; control censoring.
Explain Regression Discontinuity Design and propose a real-world scenario
Compare units just above and below a threshold for local effects; propose scenario with forcing variable.
How would you estimate causal impact using a quasi-experimental method?
DiD with Canada versus Australia; assert parallel trends; validate with pre-period plots and placebo tests.
What is Simpson's Paradox and how can it bias A/B tests?
Tests whether you recognize that aggregate trends can reverse within subgroups. A strong answer defines the paradox, gives an A/B example where treatment wins overall but loses in every segment due to skewed allocation, and prescribes stratified analysis.
How would you validate that search users are more likely to purchase?
This tests correlation versus causation in product analytics. A strong answer defines the purchase window, matches searchers to similar non-searchers, and picks a statistical test. A red flag is running a t-test without controlling for user intent or time.
Design a system to measure technical content deflection
Tests causal attribution of content on tickets. Answers must unify identity, correlate topics with NLP, compute deflection as self-service resolutions over total intent, and validate outcomes.
Design an A/B test separating novelty from true long-term impact
Tests distinguishing novelty from stable effects. Strong answer: staggered rollout with difference-in-differences comparing early and late adopters over weeks. Red flag: extending the A/B test without modeling time-interaction or control maturation.
Design an experiment and logging to link API latency to engagement
Causal inference and data integration. Randomly inject latency for a treatment group with a control at baseline, then join server trace IDs to client events via a shared request ID. Never confuse correlation with causation or miss join issues.
Why is user-level randomization flawed by spillover and what is an alternative?
Network interference. A/B tests violate SUTVA when treatment spills across edges, biasing effects. Cluster randomization uses network clusters and Horvitz-Thompson weighting for unbiased estimates.
How would you validate a feature's conversion impact given self-selection bias?
Tests causal inference for opt-in features. Strong answers use quasi-experiments like propensity matching or diff-in-diff to compare similar users and verify pre-trends. Red flag: a raw t-test between adopters and non-adopters ignoring selection bias.
How would you determine if Feature X causally drives higher retention?
Tests causal inference intuition for product metrics. Great answers propose a randomized holdback or instrumental variable, control for user intent, and estimate a local average treatment effect.
Stakeholder claims feature usage drives retention. How do you validate?
Probe confounders, reverse causality, selection bias; propose diff-in-diff or propensity scores; plot lagged usage vs retention.
When is an A/B test not feasible, and what is DiD?
This tests your grasp of causal inference when randomization isn't possible. Explain a scenario like a state-level launch, introduce Difference-in-Differences (DiD), and state its core parallel trends assumption.
Design an experiment to isolate long-term impact from novelty effect
Tests if you can design experiments for long-term impact, not just short-term lift. A good answer involves a long-running test, segmenting users by tenure, and modeling the effect over time to find its stable asymptote.
Why is user-level randomization flawed by spillover effects?
Tests your grasp of SUTVA violations in network experiments. Explain how spillover contaminates the control group, then propose graph cluster randomization—grouping users and assigning entire clusters to A/B variants—to minimize interference.
When is A/B testing not feasible, and what is an alternative?
Tests your grasp of causal inference when randomization isn't possible. A great answer names a scenario (like a regional launch), proposes Difference-in-Differences (DiD), and explains its core 'parallel trends' assumption.
How do you measure impact while accounting for the novelty effect?
Tests your ability to design experiments that isolate long-term effects. A good answer proposes a long-running A/B test, analyzing user cohorts by join date to see if initial lift decays. A red flag is ignoring the novelty effect and suggesting a short test.
Handling spillover effects in social network A/B tests
This tests your grasp of SUTVA violations in networked experiments. A great answer explains how user-level randomization causes spillover, then proposes graph cluster randomization to assign entire communities to treatment or control, minimizing…
How would you validate a PM's claim about feature success?
This tests your grasp of causal inference vs. correlation. A great answer acknowledges self-selection bias, proposes a quasi-experimental method like Propensity Score Matching to create a synthetic control group, and then compares outcomes for the matched…
Causal Impact: Measuring Effects Without an A/B Test
Causal Impact estimates an intervention's effect by modeling a 'what if' counterfactual. It's used to measure lift from a new feature or ad campaign when a clean A/B test isn't possible.
Propensity Score Matching: Simulating a Randomized Trial
Propensity Score Matching creates a 'fair' comparison from observational data, mimicking a randomized trial. It's used to estimate a treatment's effect by matching treated individuals with similar untreated ones.
SUTVA: The Assumption That Your Treatment Isn't Leaking
SUTVA assumes your treatment on one person doesn't spill over to affect others, and that the treatment is consistent for all. It's a key assumption for A/B tests, but is violated when one person's vaccine protects their unvaccinated neighbor.
Uplift Modeling: Who to Target, Not Just Who Will Convert
Uplift modeling finds who to target by predicting the *change* in behavior from an action, not just the final outcome. It's used in marketing to decide who gets a discount, optimizing spend. The footgun is confusing it with a simple conversion model.
Marketing Mix Modeling (MMM): Isolating Marketing's Impact on Sales
Marketing Mix Modeling (MMM) statistically links aggregate marketing efforts to sales outcomes over time. It's used to determine the ROI of past campaigns, like a TV ad blitz. The main footgun: the model is only as good as the historical data you feed it.
Get Causal inference bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.